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To  describe  the  organizational  conditions  under  which  group  problem  solving  is 
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decision  making  styles;  to  describe  the  effect  of  each  phase  in  the  systems 
acquisition  process  on  participatory  decision  making  styles;  and  to  identify 
the  appropriate  techniques  for  amalgamating  group  opinion. 


STUDY  REPORT  ABSTRACT: 

The  purpose  of  this  project  was  to  investigate  the  conditions  which  favor  use  of 
group  problem  solving  and  decision  making  to  examine  the  effect  of  differing 
organizational  climates  during  the  various  phases  of  the  systems  acquisition 
process  on  selection  of  appropriate  participatory  decision  making  styles,  to 
evaluate  and  identify  quantitative  methods  to  assist  in  amalgamating  group 
opinion,  and  to  understand  the  appropriate  steps  in  utilizing  this  decision 
making  approach  in  the  systems  acquisition  process.  A literature  survey  was 
conducted  of  participative  decision  making  and  of  quantitative  methods  for 
amalgamating  group  opinion. 

This  project  shows  that  it  is  possible  to  list  a set  of  questions,  the  answers 
to  which  assist  the  principal  decision  maker  in  selecting  an  appropriate  par- 
ticipatory style.  It  shows  it  is  possible  to  develop  a system  of  techniques 
useful  in  amalgamating  opinions.  It  also  shows  that  it  is  possible  to  list  the 
steps  which  a principal  decision  maker  may  follow  in  his  decision  process  when 
utilizing  group  problem  solving.  A decision  process  flow  chart  and  a decision- 
flow  diagram  are  included  in  the  report  to  aid  in  selecting  an  appropriate 
participatory  style  and  in  selecting  an  appropriate  technique  for  amalgamating 
opinion . 
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This  study  project  report  represents  the  views,  conclusions  and  recommen- 
dations of  the  author  and  does  not  necessarily  reflect  the  official 
opinion  of  the  Defense  Systems  Management  College  or  the  Department  of 
Defense. 


EXECUTIVE  SUMMARY 

The  primary  aim  of  this  report  is  to  acquaint  the  reader  with 
the  conditions  favoring  use  of  group  problem  solving  and  decision 
making  and  to  provide  some  techniques  with  which  principal  decision 
makers  or  managers  may  amalgamate  group  opinion  where  it  is  crucial 
in  reaching  an  effective  decision.  The  techniques  are  tailored  to 
the  requirements  of  a principal  decision  maker  located  in  a program/ 
project  management  office  (PMO) . This  report  examines  the  changing 
organizational  environment  as  the  program  moves  through  the  phases  of 
the  systems  acquisition  process.  The  changing  environment  is  related 
to  group  problem  solving  and  decision  making  with  the  finding  that 
the  more  participatory  styles  are  most  effective  during  the  conceptual 
and  demonstration  - validation  phases.  The  lesser  participatory 
styles  are  more  relevant  during  the  full-scale  engineering  development 
phase.  While  the  guidelines  were  tailored  for  the  principal  decision 
maker  in  the  PMO,  the  findings  have  a wider,  more  general  application. 
The  quantitative  aspects  of  the  report  may  be  omitted  without 
sacrificing  understanding  of  the  other  features.  Conversely,  the 
quantitative  techniques  described  are  valid  for  any  situation  in  which 
opinions  of  multiple  experts  require  amalgamation,  and  these  techniques 
may  be  considered  at  the  exclusion  of  the  other  features. 
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INTRODUCTION 

Purpose 

The  intent  of  this  paper  is  to  provide  the  members  of  the  systems 

acquisition  community  with  an  understanding  of  the  conditions  which 

favor  the  use  of  group  problem  solving  and  decision  making.  Another 

specific  objective  is  to  provide  techniques  by  which  program  members  can 

amalgamate  group  opinion  where  such  amalgamation  is  crucial  in  reaching 

an  effective  decision.  In  a study  of  the  decision  making  process  of 

2 

weapon  systems  acquisition,  Holland  (31:3)  quoted  then  Deputy  Secretary 
of  Defense  Packard  as  stating,  "Improvement  in  the  execution  of  these 
programs  will  be  made  to  the  extent  the  Services  are  willing  and  able  to 
improve  their  management  practices."  Although  the  quote  was  made  during 
the  Spring  of  1970,  it  is  still  true  today. 

Specific  Goals 

| 

The  specific  goals  of  this  project  are  (1)  to  describe  the  organiza- 
tional conditions  under  which  group  problem  solving  is  appropriate,  (2) 
to  specify  the  variables  which  favor  greater  or  lesser  participatory 

2 This  notation  will  be  used  throughout  the  report  for  sources  of 
quotations  and  major  references.  The  first  number  is  the  source  listed  in 
the  cited  reference  portion  of  the  bibliography.  The  second  number  is  the 
page  in  the  reference. 
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decision  making  styles,  (3)  to  describe  the  effect  of  each  phase  in  the 


system  acquisition  process  within  which  the  problem  situation  is  occur- 
ring, and  (4)  to  identify  the  appropriate  techniques  for  amalgamating 
group  opinion. 


Definitions 

The  reader  is  encouraged  to  review  the  List  of  Symbols  (page  vi)  and 
the  terms  and  their  meanings  found  in  Appendix  A,  Glossary,  before 
proceeding  further. 


Application 

There  are  no  "cookbook"  solutions  to  management  and  to  decision 
making.  However,  it  is  believed  that  a systematic,  objective  approach  to 
diagnosing  the  situation  and  applying  appropriate  participatory  style  and 
quantifying  techniques  will  aid  the  principal  decision  makers  within  the 
Program/Project  Management  Office  (PMO)  in  making  effective  decisions 
relative  to  the  systems  acquisition  process.  Rose  Glubin,  a Department  of 
Defense  manager  writing  in  the  Defense  Management  Journal,  has  asserted 
that  "the  science  of  statistics  is  not  an  alternative  to  experienced 
judgement;  rather  it  aids  and  supplements  by  creating  a more  informed 
judgement  based  on  objective  data,  principles,  and  techniques"  (24:49). 
Because  the  PMO  environment  is  becoming  increasingly  complex,  the 
individual  principal  decision  maker  is  often  unable  to  make  effective 


alone  does  not  have  all  the  technical  information  and  expertise  to  solve 
today's  problems,  neither  do  any  one  of  his  subordinates.  Fischer  (7:12) 
has  shown  that  amalgamated  opinion,  obtained  by  consolidating  and 
averaging  the  assessments  of  individuals,  typically  out  performs  almost 
all  individual  assessments.  Although  the  audience  for  whom  this  paper  is 
tailored  are  the  principal  decision  makers  within  a PMO,  the  basic  con- 
cepts expressed  have  a wider,  more  general  application  as  well. 


Scope  of  the  Project 


The  project  will  investigate  group  problem  solving  and  decision 
making.  It  will  also  evaluate  amalgamating  techniques.  This  investiga- 
tion will  be  a literature  search  utilizing  material  principally  from 
the  Defense  Systems  Management  College  Library,  The  Defense  Documentation 
Center,  and  The  Defense  Logistics  Studies  Information  Exchange. 


Limitations 


The  results  of  this  project  will  be  based  upon  information  found 
within  the  literature  referenced  in  the  Bibliography.  This  is  an  initial 
investigation  to  tie  together  separate,  yet  related  work  by  many  authors. 
Validation  of  the  concepts  proposed  within  this  paper  will  not  be  under- 
taken but  is  considered  a fruitful  follow-on  effort.  The  reader  does  not 
have  to  be  qualified  in  operations  research  methodology  to  follow  the 
discussion  and  examples  in  this  paper;  however,  he  should,  as  a pre- 
requisite, understand  the  basic  premises  of  statistics.  Mathematical 
proofs  will  not  be  included  since  this  paper  is  application  oriented. 


3 


I 


The  reader  desiring  proof  of  formulae  is  encouraged  to  avail  himself  of 
the  references  found  in  the  Bibliography.  The  reader  wishing  to  skip  the 
quantitative  aspects  of  this  report  may  do  so  and  still  gain  an  apprecia- 
tion of  group  problem  solving  and  decision  making  in  a PMO.  Conversely, 
a reader  seeking  techniques  for  amalgamating  opinion  may  exclude  the 
other  features. 


Organization  of  the  Report 


The  description  of  related  research  in  Section  II  provides  both  the 
cornerstone  and  the  foundation  upon  which  this  report  is  built. 

Appendices  B through  E complement  Sections  II  and  III  for  the  reader 
interested  in  amalgamating  techniques.  Section  II  also  provides  the 
background  discussion  which  unfolds  into  use  of  a decision  process 
flowchart  to  aid  the  principal  decision  maker  in  selecting  appropriate 
participatory  styles.  Section  III  builds  upon  the  discussion  of  Section 
II,  provides  a decision  flow  diagram  for  use  in  selecting  an  amalgamating 
technique,  and  suggests  a sequence  of  steps  which  the  principal  decision 
maker  may  follow  in  the  decision  process.  Finally,  the  report  closes  by 
making  some  concluding  remarks  and  recommending  some  areas  for  additional 
research . 


SECTION  II 


REVIEW  OF  CURRENT  LITERATURE  AND  RELATED  RESEARCH 

General 

This  section  addresses  the  research  findings  and  methodologies  that 
are  pertinent  to  the  objectives  of  this  paper.  The  libraries  and  data 
banks  are  rich  with  associated  research;  however,  no  research  was  found 
tying  together  the  separate,  yet  related  areas  of  participative  management 
or  group  problem  solving  and  the  quantitative  techniques  to  assist  in 
amalgamating  opinion.  Furthermore,  these  areas  have  not  adequately  been 
related  to  the  organizational  environment  of  the  PMO. 

Conditions  Which  Favor  Group  Problem  Solving 
and  Decision  Making 

Group  problem  solving  and  decision  making,  usually  called  participative 
decision  making  (PDM) , received  extensive  favorable  attention  in  management 
literature  in  the  late  forties  and  throughout  the  fifties  and  to  a lessor 
degree  during  the  sixties  and  seventies.  Military  applications  of  this 
style  of  management  or  decision  making  process  were  also  widely  studied  as 
is  evidenced  by  a proliferation  of  papers  written  by  students  attending 
various  service  schools  up  through  and  including  the  senior  service 


colleges  during  the  sixties  and  seventies.  The  early  writings  described 
PDM  as  being  highly  desirable  because  the  involvement  of  organization 


members  in  the  decision  process  led  to  increased  morale,  motivation,  and 
productivity.  Later  work,  while  still  indicating  PDM  can  be  effective, 
does  not  categorize  it  as  a panacea.  The  later  findings  suggest  the 
application  of  PDM  only  after  careful  scrutiny  and  determination  that 
the  organizational  environment  possesses  variables  conducive  to  its 
successful  employment  (32:4). 

The  organizational  environment  which  favors  use  of  PDM  is  one  in  which 
some  or  all  of  the  following  variables  exist: 

(1)  The  decision  is  concerned  with  complex  system(s)  (13:190); 

(2)  The  interpersonal  and  intergroup  relationships  are  not 
conflict-free  (13:319  & 27:7); 

(3)  The  decisions  are  associated  with  diverse  tasks  involving 
rapidly  changing,  and/or  unpredictable  technology  (32:5); 

(4)  The  decision  to  be  reached  is  not  overly  constrained  in 
time  (26:19  & 27:3-4); 

(5)  Others  have  data  or  knowledge  not  possessed  by  the  principal 
decision  maker  (27:7);  and 

(6)  Commitment  (not  merely  compliance)  is  required  and  cannot, 
with  a high  probability,  be  expected  to  be  achieved  if  the  decision  is 
made  solely  by  the  principal  decision  maker  (27:7). 

Another  consideration  in  examining  conditions  favoring  PDM  is  the 
question  of  the  quality  requirement  (see  Glossary).  In  other  words, 
does  it  make  a difference  to  the  principal  decision  maker  which 
alternative  is  adopted?  This  one  variable  may  override  the  use  of  the 
more  participatory  extremes  of  PDM  if  the  principal  decision  maker 


; 
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perceives  possible  goal  incongruence  between  the  goals  of  the  group  he 


might  look  to  for  PDM  and  the  organization's  goals  as  he  sees  them.  He 
might  be  reluctant  to  relinquish  his  prerogatives  when  there  is  a strong 
possibility  that  the  group's  selected  alternative  would  not  be  compatible 
with  his  view  of  the  organization's  objectives  for  decisions  he  considers 
extremely  important  (27:5). 

During  the  consideration  of  conditions  which  favor  PDM,  interaction 
factors  must  also  be  considered  since  they  may  affect  the  effectiveness 
of  the  group's  performance.  Assuming  that  the  variables  which  were 
previously  listed  favor  PDM,  Hackman  (10:27-28)  has  postulated  that  there 
are  three  interim  criteria  for  which  there  may  be  process  losses  or  pro- 
cess gains.  He  concludes  that  interventions  may  be  made  which  will 
improve  group  internal  processes  by  focusing  on  that  criteria  of  special 
saliance  to  the  kind  of  decision  upon  which  the  group  is  working.  Con- 
versely, intervention  by  the  principal  decision  maker  or  a non-salient 
aspect  of  the  group  process  could  direct  the  attention  and  energy  of  group 
members  away  from  the  issues  of  importance  and  toward  phenomena  of  little 
impact  on  eventual  group  effectiveness.  He  lists  some  possible  process 
gains  and  losses  related  to  the  interim  criteria  effort,  knowledge  and 
skill,  and  performance  strategy  as  follows: 

1.  When  effort  is  salient: 

Process  loss:  Members  fail  to  coordinate  their 
efforts  in  applying  them  to  the  (decision)  task, 
resulting  in  a "coordination  decrement." 

Process  gain:  Members  develop  strong  commit- 
ment to  each  other  and  to  the  group  that 
increases  the  amount  of  effort  they  are  willing 
to  expend  in  task  work. 
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2.  When  knowledge  and  skill  are  salient: 

Process  loss:  The  group  imperfectly  assesses 
and  weighs  the  inputs  of  members  who  have 
differential  task-relevant  talent. 

Process  gain:  Members  share  uniquely  - held 
knowledge  and  skill,  and  cooperate  to  gain  new 
learning  — thereby  increasing  the  total  pool 
of  talent  available  to  the  group. 

3.  When  performance  strategy  is  salient: 

Process  loss:  Members  imperfectly  assess  task 
requirements,  and  implement  task  - inappropriate 
strategies . 

Process  gain:  Members  invent  new  creative  ways 
of  proceeding  with  work  on  the  task.  (10:27) 

Thus,  it  is  important,  once  a decision  has  been  made  to  employ  the 
fully  participatory  style  of  PDM,  to  not  indiscriminantly  intervene  in 
the  group's  decision  process.  Otherwise,  the  interaction  of  the  group 
may  be  adversely  affected. 

While  the  preceding  discussion  indicates  that  the  group's  effective- 
ness may  be  affected  by  the  principal  decision  maker's  intervention,  it 
also  indicates  that  concerted  action  on  the  part  of  the  group  can 
influence  actions  taken  by  the  principal  decision  maker.  Richards  and 
Greenlaw  have  developed  a systems  view  of  work  group  behavior  in  terms 
of  inputs  and  outputs  which  they  assert  are  dynamically  and  reciprocally 
interdependent  in  an  open  systems  sense  (18:207-209).  Management 
decisions  in  their  model  would  relate  to  decisions  of  the  principal 
decision  maker  as  defined  in  this  paper.  A schematic  model  of  their 
systems  view  is  shown  in  Figure  1. 
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Figure  1.  Work  Group  Behavior:  A Systems  View 


(18:208) 

These  variables,  factors,  and  the  pertinence  of  intervening  actions 
are  considerations  which  must  be  examined  in  exploring  whether  or  not 
conditions  favor  use  of  PDM.  The  group,  if  used,  must  be  encouraged  and 
helped  to  develop  and  mature  into  an  effective  unit.  According  to  Schein 
(20:96)  if  the  principal  decision  maker  ignores  these  variables,  factors, 
and  the  pertinence  of  intervening  actions,  he  may  "land  in  the  trap"  of 
expecting  people  called  together  to  be  able  to  automatically  perform  as 
an  effective  psychological  group.  Further,  he  asserts,  if  they  fail  to 
perform,  he  may  then  erroneously  conclude  that  groups  are  ineffective.  He 
would  then  be  failing  to  realize  he  had  only  an  aggregate  of  people  — 
not  a group. 
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Finally,  in  examing  conditions  favoring  PDM,  consideration  is  given  to 
the  spectrum  of  possible  participation  the  principal  decision  maker 
may  desire  to  obtain  from  his  subordinates  (referred  to  in  this  paper  as 
principal  decision  maker  subordinates).  This  can  be  cast  in  a matrix, 
adapted  from  the  work  of  Weiss,  and  is  shown  in  Figure  2.  The  columns 
have  arbitrarily  been  numbered  one  to  five,  with  the  higher  numbers 
equating  to  a lessoning  of  authority  on  the  part  of  the  principal  decision 
maker  and  an  increase  in  the  freedom  subordinates  have  in  the  decision 
making  process.  The  five  style  categories  can  be  described  as  follows 
(27:3  & 26:18-19): 

1 - This  style  is  characterized  by  completely  independent  action 
on  the  part  of  the  principal  decision  maker,  using  whatever  information 
is  available  to  him  at  that  time. 

2 - The  principal  decision  maker  resolves  a situation  alone,  but  he 
obtains  any  necessary  information  of  a specific  nature  from  his  subordinates 
before  making  the  decision  himself.  He  may  or  may  not  divulge  the  purpose 
of  the  questions  he  asks  in  gathering  data  he  needs.  In  any  case,  the 
people  from  whom  he  seeks  data  play  no  role  in  analyzing  the  problem 

or  making  the  decision. 

3 - Under  this  style,  the  situation  is  explained  to  each  person  from 
whom  data  is  solicited.  Although  they  provide  data,  the  principal  decision 
maker  makes  the  decision.  The  decision  may  or  may  not  relfect  the  in- 
fluences of  those  who  have  been  consulted. 

4 - This  style,  like  style  3,  is  a variant  of  the  consultative  process. 
The  principal  decision  maker's  subordinates  are  assembled  in  a group  meeting 
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at  which  the  situation  is  explained,  data  provided,  and  alternatives  pro- 
posed. Again,  the  final  decision  is  that  of  the  principal  decision  maker 
and  may  or  may  not  reflect  the  group's  influence. 


5 - This  style  corresponds  with  Norman  Maier's  concept  of  group 
decision  making  (15:217-75)  in  which  the  principal  decision  maker's  role 
is  that  of  a chairperson  of  a group  meeting  aimed  at  reaching  consensus. 
The  principal  decision  maker  facilitates  the  flow  of  data  while  the  group 
members  try  to  reach  a consensus  as  a solution  to  the  situation.  The 
principal  decision  maker  may  or  may  not  try  to  influence  group  consensus. 
In  any  case,  he  accepts  and  implements  whatever  conclusions  the  group 
reaches . 

Phases  of  The  Systems  Acquisition  Process  and  Their  Relation 
to  Group  Problem  Solving  and  Decision  Making 


The  results  from  a recent  study  of  the  organizational  climate 
within  a number  of  Air  Force  System  Program  Offices  (SPOs)  has  a 
bearing  on  the  use  of  PDM.  While  one  style  may  work  for  some  people 
because  of  their  personality  it  may  not  for  others,  and  the  individual 
situation,  such  as  situations  with  great  time  pressures,  may  dictate 
the  participatory  style  (27:3). 


In  essence,  the  amount  of  participation  the  principal  decision 
maker  subordinates  may  be  called  upon  to  provide  will  increase 
or  decrease  in  accordance  with  the  guide  displayed  in  Figure  3. 


Figure  3. 


A Guide  for  Changing  participation  Levels 


T.F 


Participation  Should  But  Only  IF 


1.  There  is  a high 
quality  requirement. 


Increase  Others  have  data 

needed  for  a high 
quality  decision. 


2.  The  situation  is 
unstructured. 


Increase  Others  have  data 

needed  for  a high 
quality  decision. 


3.  There  is  low  goal 
congruence. 


Decrease 


There  is  strong 
quality  requirement. 


4.  There  is  a high  level  Increase 

of  commitment  (not  merely 
compliance)  required. 


Others  would  not  be 
committed  to  a 
decision  made  by 
the  leader  alone. 


5.  There  is  a high  Decrease  NO  CONDITIONS 

probability  of  com- 
mitment to  a decision 
made  by  the  leader  alone. 


6.  There  is  a high  level  of  Increase 

conflict. 


Others  would  not  be 
committed  to  a 
decision  made  by 
the  leader  alone. 


7.  There  is  a strong  Increase  NO  CONDITIONS 

need  for  training 
or  team  building. 


(27:7) 


The  variables  described  in  the  previous  subsection  can  be  addressed 
in  a decision  process  flowchart  adapted  from  Vroom  (26:19).  See  Figure  4. 
The  payoff  of  the  tree  is  a feasible  set  of  styles  which  may  be  used. 

Vroom  indicates  that  the  validity  thus  far  documented  has  been  65% 
successful  decisions  when  the  style  was  from  the  feasible  set 
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(35%  unsuccessful  when  a choice  of  style  was  made  from  the  feasible  set) 
and  29%  successful  decisions  when  the  style  was  outside  the  feasible  set 
(71%  unsuccessful  when  a choice  of  style  was  other  than  those  included  in 
the  feasible  set).  The  important  point  to  recognize  is  that,  for  most 
situations,  there  is  a feasible  set  or  range  of  styles  which  are 
appropriate  and  the  principal  decision  maker  does  not  have  to  search  for 
the  "perfect  style"  of  participatory  decision  making  (27:8). 

There  are  five  phases  of  the  systems  acquisition  process.  Since  it 
is  presumed  the  reader  is  familiar  with  these  phases,  they  will  be 
described  in  summary  fashion.  The  first  phase  is  the  conceptual  phase. 
The  primary  objective  of  this  phase  is  to  identify  those  system  concepts 
which  warrant  further  development  to  satisfy  a valid  mission  element 
need.  The  primary  efforts  within  the  PMO  during  this  phase  are  as 
follows:  (1)  Identify  and  systematically  explore  and  develop  alternative 
system  concepts,  or  modifications  to  an  existing  capability;  (2)  Develop 
the  acquisition  strategy,  including  development  of  the  Program  Master 
Plan;  (3)  Perform  cost/schedule  estimates  which  are  to  be  used  for 
comparison  of  alternatives  and  for  providing  initial  cost  and  schedule 
information  for  the  proposed  program;  (4)  Translate  operational/mission 
element  needs  into  system  technical  performance  requirements;  (5)  Develop 
logistic  and  maintenance  concepts  for  alternative  systems;  (6)  Perform 
trade-off  analysis  between  cost,  schedule,  and  technical  performance 
alternatives;  and  (7)  Analyze  alternate  procurement  and  management 
approaches,  including  procurement  and  business  planning  (33:9). 


(Adapted  from  26:19) 


The  second  phase  is  the  demonstration/validation  phase.  During 
this  phase,  the  primary  objective  is  to  conduct  demonstrations  of  viable 


alternative  systems  or  critical  subsystems  and  to  provide  a basis  for 
selection  of  a system  for  full  scale  engineering  development.  The  act- 
ivities of  the  PMO  are  centered  around  the  following:  (1)  Develop 
alternative  system(s)  design(s);  (2)  Establish  relationships  between 
need,  urgency,  risk  and  worth  to  permit  practical  trade-offs  among  cost, 
schedule,  and  technical  performance;  (3)  Reduce  or  solve  the  risk  of  the 
proposed  system  through  the  conduct  of  competitive  demonstrations  of  full- 
scale  prototypes  of  systems  or  critical  subsystems  in  realistic  environ- 
ments, when  feasible  and  practical,  and  alternatively  through  detailed 
studies  and  analyses;  and  (4)  Expand  and  refine  the  acquisition  strategy 
(33:9-10). 

During  the  next  phase,  the  full  scale  engineering  development  phase,  the 
primary  objective  is  to  complete,  prior  to  commitment  to  full  scale  pro- 


duction, the  design,  the  assembly,  and  the  testing  of  the  system  and  its 
support  requirements  to  ascertain  whether  the  required  operational 


capability  can  be  achieved  subject  to  the  imposed  cost  constraints. 
Normally,  the  physical  size  of  the  PMO,  in  terms  of  number  of  people, 
expands  during  this  phase  to  be  able  to  perform  the  planning  of  future 
requirements  (engineering  and  production  design,  test  and  evaluation, 
and  required  training  and  support  planning),  procuring  long  lead  time 
items,  administering  the  numerous  contracts,  managing  the  on-going 
program,  and  complying  with  the  reporting  procedures  of  higher 
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authorities  (33:13-14). 


The  fourth  of  the  five  phases  is  the  production  phase.  The  primary 
objective  of  this  phase  is  to  produce  and  deliver  operational  systems 
(including  all  necessary  items  of  logistic  support)  for  deployment  by  the 
operational  forces.  The  tasks  in  which  the  PMO  personnel  are  involved 
are  as  follows:  (1)  Production  management  of  the  weapon  system,  training 
equipment,  spares  and  support  equipment,  including  monitoring  of  con- 
tractor progress  and  performance,  resolving  encountered  difficulties,  and 
controlling  schedules  and  costs;  (2)  Implementation  of  logistic  support 
activities,  such  as  provisioning;  and  (3)  Commencing  preparation  for 
transitioning  and  turning  over  management  responsibilities  for  the 
system,  if  applicable  (33:14-16). 

The  fifth  phase  is  deployment.  Its  primary  objective  is  to  provide 
systems  and  support  to  operational  units.  This  period  commences  with 
user  acceptance  of  the  first  operational  unit  and  ends  with  the  systems 
phase-out  of  the  inventory.  As  far  as  the  PMO  is  concerned,  transition 
terminates  most  of  the  program  management  responsibilities  and  the  size 
of  the  office  shrinks  to  a size  sufficient  to  accomplish  residual  tasks 
which  remain.  Again,  the  transition  process  and  transfer  of  responsibility 
to  a logistic  or  readiness  command  is  service  dependent;  however,  in  most 
cases,  the  PMO  is  eventually  disbanded  (33:16).  While  the  foregoing 
description  of  the  phases  of  the  systems  acquisition  process  has  been 
brief,  assuming  reader  familiarity,  it  should  be  sufficient  for  even 
those  less  conversant  in  the  life  cycle  of  system  acquisitions  to  under- 
stand the  following  comments. 
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In  a study  of  the  PMO  organizational  climate  during  the  various 


phases  of  the  systems  acquisition  process,  Larson  and  Ruppert 
examined  13  Air  Force  SPOs,  which  will  be  assumed  for  the  purpose  of 
this  paper  to  be  representative  of  PMOs  of  all  services.  They  used 
an  instrument  developed  by  Rensis  Likert  of  the  Institute  for  Social 
Research  at  the  University  of  Michigan,  the  "Likert  Profile  of  Organiza- 
tional Characteristics,"  to  determine  the  nature  of  the  organizational 
climate.  It  uses  questions  to  measure  individual  member  perceptions 
of  the  organizational  climate.  The  climate  is  measured  along  a continuum 
from  negative  (System  1),  in  which  all  decisions  are  made  by  top  management, 
communications  flow  only  downward,  and  personnel  within  the  organization 
do  not  contribute  to,  or  reject,  the  organization's  goals;  to  positive 
(System  4),  in  which  the  decision  making  process  is  shared,  communications 
flow  not  only  downward,  but  upward  and  horizontally,  and  the  individuals 
accept  the  goals  of  the  organization,  adopt  these  as  their  own,  and 
contribute  toward  achieving  these  goals  (33:19-20).  They  concluded  that 
the  System  Program  Offices  have  a tendency  to  more  nearly  practice  a 
participative  style  (System  4)  during  the  conceptual  and  demonstration/ 
validation  phases  than  in  any  other  phase.  As  the  program  progressed 
into  full  scale  engineering  development,  they  found  that  individual 
perceptions  of  the  organizational  climate  changed.  They  attributed  the 
change  to  organizational  expansion  which  results  in  the  organization's 
moving  towards  a System  1 style  of  management.  When  System  4 management 
is  no  longer,  or  can  no  longer,  be  practiced,  they  deduced  that  individuals 
have  a lesser  tendency  to  identify  with  the  organization  goals  and  to 
replace  them  with  their  individual  goals.  They  found  that,  as  the  program 
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moves  into  the  final  two  phases,  it  was  interesting  to  note  that,  instead 


of  the  organization  becoming  more  structured,  the  climate  reverses  its 
trend  and  progresses  back  toward  System  4.  However,  they  observed  that 
it  never  reaches  the  participative  management  level  attained  during  the 
first  two  phases  (33:57-58). 

Although  the  Larson  and  Ruppert  study  used  a range  of  styles  from  one 
through  four  while  this  paper  uses  a continuum  containing  five  levels, 
the  point  to  be  made  is  that  generally  the  climate  is  more  favorable  for 
the  more  participatory  styles  of  decision  making  during  the  conceptual 
and  demonstration/validation  phases,  to  a lesser  degree  during  the  pro- 
duction and  deployment  phases,  and  least  favorable  during  the  full 
scale  engineering  development  phase. 

Methodologies  for  Amalgamating  Opinions 

The  principal  decision  maker  may  require  tools  to  assist  in  combining 
the  opinions  of  those  he  calls  upon  to  assist  him  in  reaching  a decision. 

The  process  of  combining  these  opinions  into  a single  joint  order  is  the 
function  of  amalgamation.  Those  to  whom  he  looks  to  for  information  have 
been  called,  in  the  earlier  sub-sections,  principal  decision  maker  sub- 
ordinates. Throughout  this  sub-section,  they  will  be  referred  to  as  judges. 
This  terminology  is  consistent  with  the  majority  of  the  literature  on  the 
subject  of  amalgamating  opinion,  and  is  used  here  as  a somewhat  wider 
label  to  encompass  those  individuals  the  principal  decision  maker  may  look 
to  for  information  but  who  are  not  his  true  subordinates.  Because  the 
principal  decision  maker  is  not  going  to  be  receptive  to  learning  a large 
number  of  different  techniques  to  assist  in  reaching  his  decision,  the 
following  constraints  were  imposed  for  the  purpose  of  evaluating  techniques 
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to  be  recommend  .1  in  this  report: 

(1)  The'e  must  be  only  a small  number  of  techniques  for  the  principal 
decision  maker  to  learn,  and 

(2)  Considered  collectively,  the  techniques  must  be  sufficiently 
general  to  encompass  the  wide  spectrum  of  decision  making  situations  in 
the  systems  acquisition  process. 

It  is  conceivable  that  the  techniques  which  will  be  described  could 
be  used  for  participatory  styles  two  through  five;  however,  it  would 
probably  have  its  greatest  application  for  styles  two  or  three,  and  to 
a lessor  degree  four. 

With  the  previously  listed  constraints  in  mind,  a number  of  techniques 
were  reviewed,  examined  and  tested.  Some  of  the  more  representative  of 
these  are  now  discussed.  Algorithms  and  examples  illustrating  how  these 
techniques  may  be  employed  are  included  for  three  of  the  techniques  meet- 
ing the  constraints  and  being  recommended  to  the  principal  decision  maker. 
The  same  hypothetical  example  is  used  for  Arrow's  Method  of  Majority,  for 
the  Football  Poll,  and  for  Thurstone's  Law  of  Comparative  Judgement. 

O 

Thurstone's  Law  of  Comparative  Judgement 

Thurstone  (23:19-49)  originally  published  his  "Law  of  Comparative 
Judgement"  in  1927,  initially  intending  it  for  use  in  the  field  of 


^For  the  algorithm  and  an  example  of  its  usage,  see  Appendix  D. 
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psychometrics . It  has  also  found  wide  use  elsewhere,  in  such  diverse 


fields  as  econometrics,  sociometrics,  and  utility  theory.  Within  this 
law,  there  are  five  cases.  The  method  is  permitted  to  differ  in  each  of 
these  cases  as  it  pertains  to  assumptions,  approximations,  and  degree 
of  simplification.  The  provision  of  differing  cases  makes  this  technique 
an  extremely  versatile  tool.  The  five  cases  may  be  summarized  as  follows: 

Case  1.  - Used  when  a single  judge  makes  repeated  judgements. 

Case  2.  - Used  when  rankings  are  by  different  judges. 

Case  3.  - If  the  discriminal  deviations  of  the  same  judgement  are  un- 
correlated, the  equation  may  be  simplified. 

Case  4.  - If,  in  addition  to  the  assumption  of  Case  3,  the  variances 
of  the  discriminal  deviations  are  not  subject  to  gross 
differences,  the  equation  may  be  further  simplified. 

Case  5.  - Used  in  the  simplest  version  when  equispacing,  or  equal 
discriminal  dispersions,  may  be  assumed;  the  equation  may  be  reduced  to 
its  simplest  form. 

Another  strong  attraction  of  this  technique  is  that  it  is  simple  to 
employ,  and  it  is  possible  to  cardinally  rank  the  joint  order.  However, 
it  does  require  transitive  orders  and  requires  normality  of  the  joint 
order  of  the  rankings.  Although  Guilford  (9:154-96)  presents  some  scaling 
methods  which  would  permit  normalizing  the  rankings,  this  additional 
transformation  is  not  recommended.  There  are  other  techniques  which  do 
not  require  the  joint  order  developed  from  the  ranks  assigned  to  have  a 
particular  underlying  statistical  distribution.  The  next  two  techniques 
to  be  discussed  are  amalgamating  methods  which  do  not  require  the  rankings 
to  fit  a specified  distribution. 
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[ I 

Arrow's  Method  of  Majority  and  Football  Poll  Method  of  Ranking.1^ 

[ 1 

Arrow  proposed  a simple  majority  rule  that  makes  no  assumptions  con- 
cerning the  rankings  performed  by  the  judges.  The  Football  Poll  technique 
derives  its  name  because  it  is  used  by  the  various  press  associations 
for  rating  football  teams.  These  two  techniques  are  best  described  by 
contrasting  the  features  of  one  with  those  of  the  other  (25:  xviii-xx) . 

Each  technique  has  both  advantages  and  disadvantages.  Which  is  to  say, 
that  one  is  not  necessarily  "good"  and  the  other  "bad."  Rather,  each 
technique  may  be  either  good  or  bad  dependent  solely  upon  whether  the 
conditions  under  which  it  is  employed  fit  the  assumptions  which  must  be 
made  concerning  the  data  with  which  it  is  used. 

In  using  the  Football  Poll  technique,  points  are  awarded  for  position. 

For  example,  one  point  might  be  awarded  for  first  place,  two  points  for 
second  place,  etc;  or,  as  with  the  press  rankings,  the  reverse  might  be 
used,  awarding  points  in  diminishing  order  from  first  place  to  the  lowest 
place  considered.  The  points  awarded  a particular  place  is  the  R associa- 
ted with  that  objective  (the  objective  being  a team,  a proposal,  or 
whatever  is  being  ranked).  The  objectives  are  then  ordered  on  the  basis  of 
the  Rj's.  Those  with  the  lower  sum  (or  higher  sum,  as  the  case  may  be) 
being  ordered  as  the  most  preferred.  The  problem  with  this  technique  is 

^For  the  algorithm  and  an  example  of  its  usage,  see  Appendix  B. 
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that  the  majority  of  the  judges  may  feel  a particular  objective  is  better 
than  ail  of  the  alternatives  and  yet  it  may  ultimately  be  ranked  second, 
third  or  lower.  The  reason  for  this  happening  is  that  artifical  informa- 
tion, not  necessarily  truly  representative  of  the  judge's  scalings  of  the 
objectives,  is  that  which  is  added.  The  Football  Poll  procedures  assume 
that  there  is  equal  spacing  between  preferences,  which  is  to  say,  that  the 
amount  one  objective  is  preferred  over  another  is  always  equal.  The  rank 
ordering  provided  by  the  judges  does  not  necessarily  imply  this.  It  also 
assumes  that  the  Rj  are  additive.  Therefore,  because  it  is  easy  to  apply 
and  its  results  easily  displayed,  the  Football  Poll  Method  of  Ranking  is 
widely  accepted  and  misused  by  those  not  understanding  its  assumptions. 

In  contrast  to  the  Football  Poll,  the  Arrow's  Method  of  Majority 
provides  a similar,  easily  displayed  result,  but  requires  no  assumptions 
to  be  made  concerning  the  rankings.  Each  objective  is  considered  as  an 
entity  and  the  number  of  times  it  is  preferred  to  each  other  alternative 
is  counted.  The  results  are  cast  in  an  array  which  is  then  converted  to 
a new  array  portraying  a 0,1, or  1/2,  depending  upon  which  entity  was 
preferred  less  times,  more  times,  or  an  equal  number  of  times  to  its 
complement.  For  example,  if  "a"  is  preferred  to  "b"  two  times  and  "b" 
is  preferred  to  "a"  once;  then,  "a"  is  preferred  to  "b"  by  majority  vote. 
In  this  event,  "a"  would  receive  a 1 and  "b"  would  receive  a 0. 

Later  work  by  Shannon  and  Wyatt  (34:5-10)  has  added  to  the  features 
of  Arrow's  Method  of  Majority  Rule  the  ability  to  reflect,  or  take  into 
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account,  the  differing  expertise  or  knowledge  of  the  judges  for  the  various 


entities  being  ranked. ^ 

The  greatest  limitation  of  this  technique  is  that  it  fails  to  consider 
whether  one  judge  has  a strong  preference  for  one  objective  as  opposed  to 
one  or  more  alternatives,  while  a second  judge  has  a weak  counter  pre- 
ference concerning  these  same  objectives. 

Some  Rejected  Techniques 

Certainly  all  techniques  for  quantification  of  opinion  cannot  be 
discussed  here.  However,  to  ignore  those  which  were  considered  but  to  be 
less  appropriate  would  be  tantamount  to  glossing  over  the  very  existence 
of  techniques.  These  other  techniques  are  described  in  Appendix  E.  As  was 
previously  set  forth  in  establishing  constraints  on  selecting  techniques 
to  be  recommended  to  the  principal  decision  maker,  there  must  be  only  a 
limited  number  of  techniques  for  the  decision  maker  to  learn  and  they  must 
collectively  cover  the  spectrum  of  his  applications.  Consequently,  a 
number  of  techniques  were  rejected,  not  because  of  invalidity,  but  simply 
because  they  were  less  desirable  than  those  selected  in  the  accomplishment 
of  similar  purposes. 


""’’For  the  modifying  algorithm  and  an  example  of  its  usage,  see  Appendix 

C. 
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SECTION  III 


GROUP  PROBLEM  SOLVING  TECHNIQUES  TO  ACHIEVE 
EFFECTIVE  DECISIONS 

A Process  for  Amalgamating  Opinions 

The  three  techniques  selected  in  the  previous  section  collectively 
encompass  the  wide  spectrum  of  decision  making  situations  in  the  systems 
acquisition  process.  A few  simple  questions  will  be  helpful  in  allowing 
the  principal  decision  maker  to  select  the  most  appropriate  amalgamating 
technique  for  a given  application.  The  following  questions,  used  in 
conjunction  with  the  accompanying  decision-flow  diagram  (see  Figure  5), 
are  designed  to  assist  in  this  determination: 

(1)  Do  the  judges  selected  possess  varying  degrees  of  expertise  in 
relation  to  the  sub-categories  reflected  by  the  totality  of  objectives  to 
be  ranked? 

(2)  Can  it  be  assumed  that  the  rankings  of  the  individual  judges  may 
not  be  equispaced? 

(3)  Can  it  be  assumed  that  the  R j ' s (which  are  the  sum  of  ranks 
given  by  the  set  of  m judges  for  the  j-th  objective)  may  not  be  additive? 

(4)  Is  intransitivity  a possibility  in  the  rankings,  and  conversly 
is  there  no  requirement  for  a cardinal  ranking  of  the  joint  order? 

Prior  to  making  the  final  selection  of  an  amalgamation  technique  to  be 
employed,  it  is  first  necessary  to  perform  some  statistical  tests  on  the 
data.  These  statistical  tests  must  be  performed  to  insure  the  validity  of 
the  joint  order.  Which  is  to  say,  the  agreement  of  the  judges  must  be 
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ascertained  to  be  within  acceptable  limits.  Also  the  data's  distribution 
may  have  to  be  checked. 

The  statistical  tests  to  be  performed  must  be  identical  for  most  of 
the  techniques  which  have  been  recommended  in  this  paper.  If  they  are 
not,  the  decision  maker  could  not  select  a technique  until  after  acquiring 
the  rankings  and  testing  of  the  joint  order  of  this  data.  However,  that 
would  be  at  variance  with  the  proper  order  of  events  in  which  selection  of 
the  amalgamation  technique  should  come  very  early,  before  determining 
sample  size  and  stratification,  before  developing  interrogation  methods, 
and  before  obtaining  rankings.  Therefore,  it  is  imperative  that  these 
statistical  tests  be  applicable  to  all  of  the  recommended  techniques.  An 
exception  to  the  requirement  for  statistical  tests  exists  for  Case  1 of 
Thurstone's  Law  of  Comparative  Judgements,  since  only  a single  judge  is 
involved. 

The  first  test  that  must  be  performed  is  a measurement  of  the  degree 
of  correspondence  between  two  or  more  rankings.  More  specifically,  a 
measurement  of  the  intensity  of  rank  concordance  must  be  made.  For 
correlation  of  two  judges,  or  of  a single  judge  and  a standard,  there  are 
several  tests  which  might  be  applied.  The  most  widely  accepted  of  these 
are  Kendall's  Rank  Correlation  Coefficient  TJ  (Greek  letter  tau),  and 
Spearman's  Coefficient  of  Rank  Correlation  O (Greek  letter  rho).  For 
both  of  these,  the  numerical  values  of  T and  />  range  from  +1.00  (perfect 
positive  correlation,  meaning  every  objective  has  the  same  rank  in  both), 
to  0.00  (no  correlation,  meaning  independence),  and  down  to  -1.00  (perfect 
negative  correlation,  meaning  one  ranking  is  the  inverse  of  the  other). 


‘ 
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. An  excellent  discussion  of  these  tests-  and  a comparison  of  their  uses  has 


been  made  by  Kendall  (14:3-33).  Both  of  these  are  too  limited  in 
application  to  be  of  use  to  us  in  the  amalgamation  of  group  opinion. 

To  be  sufficiently  general  for  our  use,  the  test  selected  must  handle 
not  only  two  sets  of  rankings,  but  it  also  must  handle  multiple  rankings. 
Averaging  all  the  possible  values  T”  or between  pairs  of  observers 
would  be  a solution,  but  it  would  prove  very  tedious  for  large  numbers  of 
judges.  Recognizing  this,  Kendall  developed  a Coefficient  of  Concordance 
W (14:94-116).  W measures  the  commonality  of  judgements  of  multiple 
observers.  Unlike  the  tests  of  two  rankings,  the  range  of  numerical 
values  of  W varies  from  0.00  to  +1.00,  and  not  from  -1.00  to  +1.00. 
Symmetrical  opposites  can  not  exist  in  the  sense  we  are  considering  for 
rankings  of  more  than  two  judges.  If  there  are  three  judges,  A may  dis- 
agree with  B on  a comparison  and  also  disagree  with  C,  but  then  B and  C 
must  agree.  This  reasoning  can  be  carried  on  for  four  or  more  judges 
also.  Therefore,  if  all  judges  agree,  W will  approach  +1.00  as  a limit. 

The  algorithm  for  Kendall's  Coefficient  of  Concordance  is  found  in  Appendix 
B. 

The  non-parametric  test  described  above  is  the  only  test  required  when 
using  the  Football  Poll  Technique  or  Arrow's  Method  of  Majority.  Recalling 
that  Thurstone's  Law  of  Comparative  Judgement  requires  normality  of  the 
joint  order  of  the  rankings,  when  this  technique  is  being  utilized,  a test 
for  normality  is  thus  required.  The  classical  test  for  normality  of  a 
distribution,  the  Chi-Square  Test  for  Goodness  of  Fit,  is  the  test 
recommended  for  use.  It  will  be  employed  in  addition  to  Kendall's 
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Coefficient  of  Concordance  when  Thurstone's  Law  of  Comparative  Judgement, 
Cases  2 through  5 are  being  used.  The  procedure  for  computing  the 
Chi-Square  test  statistic  is  included  in  Appendix  D. 


f 


These  tests,  when  required,  must  be  performed  to  assure  that  the 
technique  initially  selected  is  compatible  with  the  joint  order  of  the 
rankings.  If,  for  instance,  it  was  planned  to  use  Thurstone’s  Law  of 
Comparative  Judgements,  Cases  2-5,  and  the  Chi-Square  Test  fails,  it  is 
still  possible  to  use  the  data  as  submitted  by  the  judges  in  an  alternate 
technique  such  as  Arrow's  Method  of  Majority. 

Dimensions  of  Effective  Decisions 

For  a decision  to  be  effective,  two  essential  dimensions  must  be 
considered.  Maier  (15:275-77)  describes  decision  effectiveness  as  being 
a function  of  quality  and  acceptance.  He  states  that  the  quality  dimension 
is  the  degree  to  which  the  decision  represents  objective  facts,  and  the 
acceptance  dimension  is  a measure  of  how  the  group  that  must  execute  the 
decision  feels  about  the  decision  which  was  reached.  For  a decision 
to  be  effective,  both  high  quality  and  high  acceptance  are  required. 
Therefore,  the  group  problem  solving  process  must  consider  both  facts 
and  feelings.  Since  the  method (s)  of  dealing  with  the  two  are  not 
necessarily  the  same,  the  principal  decision  maker  must  be  skillful  in 
making  diagnostic  judgements  and  then  dealing  with  each.  The  effectiveness 
of  the  decision  can  be  described  by  the  equation 
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ED  = Q x A,  where 


( 


ED  = Effective  decision 
Q = Quality  (facts) 

A = Acceptance  (feelings) 

It  will  be  noted  that  Maier  used  a multiplication  sign  in  the  relationship. 
He  assumed  that  when  both  dimensions  were  positive  that  the  effectiveness 
would  be  highly  positive.  Therefore,  he  established  his  equation  in  this 
vein.  He  postulated  that  if  either  the  quality  or  the  acceptance  dimension 
is  zero,  the  effectiveness  is  zero.  He  further  felt  that  if  either 
dimension  is  negative,  the  effectiveness  will  be  negative.  Finally,  he 
asserts  that  if  both  dimensions  are  negative , the  relationship  will  still 
provide  a valid  answer  mathmetically  and  conceptually  since  it  will  display 
negative  quality  (violation  of  objective  facts)  times  negative  acceptance 
(rejection  by  the  group)  and  consequently  positive  effectiveness  (a  correct 
decision  of  non-implementation). 


The  Problem  Solving  Process 


Recognizing  the  dimensions  of  an  effective  decision,  we  can  now 
describe  the  steps  necessary  to  reach  such  a decision.  The  problem  solving 
process  is  described  in  many  management  texts,  operations  research  texts, 
and  military  manuals.  What  will  be  described  below  is  a synthesis  of  the 
ideas  from  many  sources  which  together  provide  a logical  and  orderly  means 
for  arriving  at  a solution  to  a problem  confronting  a principal  decision 
maker.  (For  the  reader  desiring  further  information  not  provided  in 


this  paper,  the  following  sources  are  suggested:  2:23-57;  12:12-14;  1:67-76; 
5:11-13;  and  21:46-58).  It  is  suggested  that  the  principal  decision  makers 
utilize  the  following  steps: 

1.  Define  and  state  the  problem. 

2.  Define  expectations. 

3.  Consider  the  organizational  environment  of  the  particular 
problem  situation. 

4.  Evaluate  and  select  an  appropriate  participatory  style. 

5.  Dependent  upon  the  selected  style,  review  information  sources 
which  are  feasible  and  suitable. 

6.  Collect  data. 

7.  If  appropriate,  amalgamate  opinions. 

8.  Develop  conclusions. 

9.  Plan  implementing  actions. 

10.  Make  decision. 

11.  Implement  decision  and  evaluate  outcome(s).  (This  may  dictate 
reentering  the  process  at  step  one). 

Section  II  and  the  earlier  portions  of  this  section  (Section  III) 
will  have  afforded  the  reader  a background  for  use  of  the  eleven  step 
process.  The  reader  will  recognize  that  Section  II  contained  both  a 
narrative  description  and  a decision  process  flow  chart  (Figure  4), 
relating  to  the  participatory  styles  appropriate  for  various  situations. 
Those  guidelines  will  aid  in  accomplishing  Step  4.  A decision-flow 
diagram,  which  was  provided  earlier  in  Section  III  (Figure  5)  will  assist 


31 


in  selecting  an  appropriate  amalgamating  technique.  This  will  facilitate 
accomplishing  Step  7. 


As  a part  of  developing  conclusions,  Step  8,  the  decision  maker(s) 
should  forecast  the  consequences  of  the  solutions  proposed,  testing  and 
evaluating  the  possible  solution  before  finalizing.  Throughout  the 
process,  the  effectiveness  of  the  decision  must  be  considered  in  terms 
of  quality  (facts)  and  acceptance  (feelings).  Neither  of  these  dimensions 
can  be  permitted  to  become  zero  or  negative  for  a highly  effective  decision 
to  result. 
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SECTION  IV 


SUMMARY 

Conclusi ons 

The  examination  and  discussion  contained  within  this  report  considered 
the  conditions  which  favor  use  of  group  problem  solving  and  decision  making, 
the  phases  of  the  systems  acquisition  process  and  their  relation  to  group 
problem  solving  and  decision  making,  techniques  available  for  amalgamating 
group  opinion,  and  the  steps  in  the  group  problem  solving  process  to 
achieve  an  effective  decision. 

This  project  shows  that  it  is  possible  to  list  a set  of  questions  the 
answers  to  which  assist  the  principal  decision  maker  in  selecting  an 
appropriate  participatory  style.  It  also  shows  that  it  is  possible  to 
develop  a system  of  techniques,  small  in  number  but  extensive  in  their 
applicability,  which  will  aid  the  principal  decision  maker  in  quantifying, 
by  ranking  ordinally  or  cardinally,  the  opinions  of  those  he  may  look  to 
for  assistance  in  his  decision  making  role.  The  report  synthesizes,  from 
a variety  of  sources,  the  steps  which  a principal  decision  maker  may  follow 
in  his  decision  process  when  utilizing  group  problem  solving.  It  is  con- 
cluded that  the  more  participatory  styles  of  decision  making  have  their 
greatest  application  during  the  conceptual  and  demonstration/validation 
phases  and  the  less  participatory  styles  are  found  in  greater  favor  in  the 
full  scale  engineering  development  phase. 
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Areas  of  Additional  Research 


It  is  recommended  that  a follow-up  project  be  undertaken  to  determine 
if  the  findings  of  the  study  of  Air  Force  SPOs  (33:56-58)  are  borne  out 
by  studies  of  PMOs  in  the  other  Services.  Research  should  also  be  under- 
taken to  validate  or  refute  productivity  improvements  and/or  decision 
effectiveness  measures  associated  with  use  of  styles  of  decision  making 
appropriate  to  the  situational  environment.  This  should  be  accomplished 
across  the  widest  possible  variety  of  PMOs  and  during  all  phases  of  the 
systems  acquisition  process. 
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APPENDIX  A 


I 


GLOSSARY 


Algorithm  - a simple  to  follow  step-by-step  direction  for  solving  a 
problem. 

Amalgamate  - to  unite;  mix;  combine. 

Attribute  - a characteristic;  composed  of  more  than  one  objective. 

Cardinal  - one  of  the  highest  of  the  measurement  scales;  numbers  are 
assigned  to  objects  in  a manner  which  provides  a representation  of 
distance  between  them. 

Certainty  - the  decision  maker  knows  precisely  the  consequence  attached 
to  every  course  of  action  available  to  him. 

Concordance  - the  mutual  interrelationship  between  more  than  two  variables. 

Con flic t - the  behavior  of  an  individual,  a group,  or  an  organization 
which  impedes  or  restricts  (at  least  temporarily)  another  party  from 
attaining  its  desired  goals  (the  consequences  may  be  beneficial  if  they 
produce  new  information  enhancing  the  decision  making  process). 

Correlation  - the  mutual  interrelationship  between  two  variables. 

Decision  - a judgement  or  a choice  between  alternatives. 

Discriminal  deviation  - difference  that  exists  between  objective  rankings. 

Goal  congruence  - a measure  of  the  agreement  of  the  interests  of 

subordinates  --  especially  implementors  --  with  the  same  interests 
and/or  goals  of  the  organization  in  resolving  the  problem  situation. 

Group  - any  manifold  of  persons,  identifiable  over  a period  of  time,  and 
sufficiently  integrated  so  that  its  actions  and  objectives  are 


- 


identifiable. 
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Group  role  - marked  by  decline  in  self-oriented  behavior  of  members  who 
are  interacting  with  one  another,  are  psychologically  aware  of  one 
another,  and  perceive  themselves  to  be  a group. 

Hypothesis  - an  unproven  theory,  tentatively  accepted  to  explain  certain 
facts;  in  contrast,  null  hypothesis,  is  that  which  is  normally  tested 
in  statistics,  the  objective  being  to  disprove  it  and  thereby  prove 
the  hypothesis. 

Individual  role  - the  premise  that,  at  least  initially,  when  two  or  more 
people  come  together  as  a group,  there  will  be  a period  of  essentially 
self -oriented  behavior  such  as  identity  selection,  power  or  influence 
distribution,  goal  congruence  examination,  and  acceptance  and  intimacy 
resolution. 

Interquartile  range  - the  middle  50  percent  of  whatever  is  being 

measured,  for  example,  responses;  so  named  because  it  defines  the 
spread  between  the  measures  standing  between  the  first  two  quarters 
and  the  last  two  quarters  of  a distribution,  called  quartiles. 

Intransitivitv  - (See  transitivity). 

Joint  order  - the  joint  set  acquired  by  combining  the  individual  orders 
of  the  various  judges. 

Judge  - the  evaluator;  expert;  respondent;  observor;  or  other  description 
of  the  person  with  expertise  in  an  area  that  performs  the  individual 
rankings . 

Mean  - a measure  of  central  tendency;  the  measure  computed  by  adding  all 
measurements  and  dividing  by  the  total  number  of  cases. 

Median  - a measure  of  central  tendency;  the  measure  that  divides  the  group 


into  halves  of  the  same  number;  the  middle  value  in  order  of  size  if  N 
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( is  odd,  or  the  mean  of  the  two  middle  items  if  N is  even. 

Normalizing  - the  process  of  placing  a group  of  values  on  a common  scale. 

Ob jective  - the  variable;  the  alternative;  the  option;  the  subject  of  the 
judge's  ranking  process. 

Order  - to  place  in  a relationship  indicating  preference,  utilizing  some 
scale  of  measurement. 

Ordinal  - the  lowest  scale  of  measurement  permitting  the  property  of  rank 
order;  if  two  objectives  are  unequal,  the  direction  of  their  inequality, 
that  is , a > b or  b > a is  indicated. 

Participative  decision  making  (PPM)  - a decision  making  process  in  which 
those  who  ill  carry  out  the  decision  have  a substantial  input  in  the 
decision  process  or  actually  make  the  decision  themselves.  Within  this 
report,  the  terms  participative  management  and  participation  will  be 
considered  synonymously  with  PDM. 

Pre fere nee  - as  contrasted  with  indifference,  the  former  indicates  a 
discernment  of  difference  in  two  objectives,  shown  as  a > b or 
b > a,  while  indifference  is  displayed  a~  b or  a=b.  Strong  preference 
permits  ordering  only  using  inequality  signs,  while  weak  preference 
allows  the  use  of  indifference  or  ties  in  ordering  to  appear. 

Principal  decision  maker  - the  manager  charged  with  responsibility  for  a 
decision  and  having  the  authority  to  make  the  decision. 

Principal  decision  maker  subordinates  - individuals  subordinate  to  the 
manager  categorized  as  the  pr  ncipal  decision  maker;  for  the  purpose 

I 
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of  this  report,  these  are  individuals  within  a program/project  manage- 
ment office  (PMO)  subordinate  to  a program  manager  or  to  a lower  level 
manager.  The  subordinate  may  or  may  not,  in  turn,  be  the  manager  or 
supervisor  of  others. 

Quality  requirement  - the  question  of  whether  or  not  it  makes  a difference 
which  course  of  action  is  adopted. 

Rank  ordering,  - listing  of  opinions  of  judges. 

Result  of  research  - a term  sometimes  used  indicating  the  conclusion 
drawn  from  the  joint  ordering;  the  making  of  the  decision  or 
recommendation. 

Risk  - the  decision  maker  does  not  know  the  definite  consequence  attached 
to  a course  of  action,  but  a probability  may  be  attached  to  every 
possible  consequence. 

Transitivity  - as  contrasted  with  intransitivity,  states  that  if  a judge 
has  any  preference  among  three  alternatives  they  must  be  consistent 
in  the  following  sense: 

If  a - b 

b ~ c , then  a c ; 
if  a > b 

b > c , then  a > c; 
and  if  a > b 

b ~ c,  then  a > c. 

Any  other  condition  is  intransitive.  That  is  a > b,  b>  c, 
can  not  permit  or  imply  c > a without  being  intransitive. 
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Uncertainty  - decision  situations  in  which  either  the  probabilities 


associated  with  the  outcomes  are  a matter  of  individual  judgement, 
or  no  judgement  at  all  can  be  made  about  them. 

Utility  - the  perceived  worth  of  an  alternative  to  a judge;  procedure 
for  identifying  preferences  explicitly. 
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APPENDIX  B 

THE  ARROW'S  METHOD  OF  MAJORITY  AND 
THE  FOOTBALL  POLL  METHOD  OF  RANKING 

Football  Pool  Algorithm. 

1.  Let  n = number  of  objectives  to  be  ranked,  and  let  m = number  of 
judges  assigning  ranks.  List  each  judge's  preferences,  then  cast 
them  in  a matrix  of  rankings.  If  several  objectives  are  tied,  give 
them  the  value  of  the  mid-point  of  the  ranks  for  which  they  are  tied 
(that  is,  if  two  objectives  are  tied  for  ranks  3 and  4,  give  them 
each  3.5;  or  if  three  objectives  are  tied  for  ranks  7,  8,  and  9, 
give  them  each  8). 

2.  For  each  objective,  determine  Rj , the  sum  of  the  ranks  assigned 
to  the  j-th  objective  by  the  m judges. 

3.  Determine  the  mean  of  the  Rj  values  by  use  of  the  relation 


4.  Express  each  Rj  as  a deviation  from  that  mean,  R.,  and  square 
these  deviations.  Sum  these  squares  to  obtain  ,4.  : 

Rj?  * (Rt  -Rj)** (Rn-Rj)3 
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5.  a.  If  there  are  few  or  no  ties  in  the  m sets  of  rankings,  calculate 
the  coefficient  of  concordance  by  the  relation 


w * 


A 

(ik)™*  (V  - n) 


> 


where  Of  w£l,  and  0 means  perfect  disagreement, 

, 1 means  perfect  agreement, 

b.  If  there  are  a considerable  number  of  ties  relative  to  the 
number  of  objectives  n in  the  m sets,  use  the  equation 


W = 


n)  - mg  T- 

-i*/ 


where 


T 


z (t‘- 1 ) 

IZ 


for  each  of  the  m sets  or  rows  having  ties; 

t = number  of  observations  in  a group  tied  for  a given  rank;  £ 
means  to  sum  over  all  the  tied  groups  within  any  one  row;  and 
means  sum  the  T's  over  all  m rankings. 

(For  example,  if  judge  1 was  indifferent,  in  ranking  two  objectives. 


or  had  one  tie;  if  judge  2 was  indifferent  in  ranking  two  objectives 


twice,  or  had  two  ties;  and  if  judge  three  was  indifferent  both  in  ranking 
three  objectives  and  indifferent  in  ranking  two  objectives;  then,  the 
correction  factor  for  ties  would  be  as  follows: 


■ --  o.ff 

12 

(2  3-  ^)  + (2*  - 2 ) 

12 

-£)_+■  (3s  -3) 

>2. 

= 0 5 1.0  + 2-  S ^ 


1.0 


4 ■) 


6.  The  test  used  to  determine  whether  the  calculated  coefficient 
of  concordance  is  significantly  different  from  zero,  (that  is,  whether 
there  is  agreement  between  the  judges  at  a given  level  of  significance), 
depends  on  the  number  of  objectives  which  were  ranked,  the  size  of  n. 

a.  If  n5  7,  the ^ calculated  in  step  4 is  used  instead  of  W. 

Use  table  1 or  more  complete  tables  found  in  (22:286). 

b.  If  n>7,  a Chi-Square  value  is  calculated  using  W in  the 
relation 

JC*  --  rn  (n  - ,)  w. 

C*/c 
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m 

n 3 

4 

5 

6 

7 

3 

64.4 

103.9 

157.3 

4 

49.5 

88.4 

143.3 

217.0 

5 

62.6 

112.3 

182.4 

276.2 

6 

75.7 

136.1 

221.4 

335.2 

8 

48.1 

101.7 

183.7 

299.0 

453.1 

10 

60.0 

127.8 

231.2 

376.7 

571.0 

15 

89.8 

192.9 

349.8 

570.5 

864.9 

20 

119.7 

258.0 

468.5 

764.4 

1.158.7 

TABLE  1.  CRITICAL  VALUES  OF/4.  AT  THE  .05  SIGNIFICANCE  LEVEL ; 
ADAPTED  FROM  (22:286). 

LEVEL  OF  SIGNIFICANCE 


d.f. 

.10 

.05 

.01 

7 

12.02 

14.07 

18.47 

8 

13.36 

15.51 

20.09 

9 

14.68 

16.92 

21.67 

10 

15.99 

18.31 

23.21 

12 

18.55 

21.03 

26.22 

14 

21.06 

23.68 

29.14 

16 

23.54 

26.30 

32.00 

18 

25.99 

28.87 

34.80 

20 

28.41 

31.41 

37.57 

TABLE  2.  CRITICAL  VALUES  OF  X*  ADAPTED  FROM  (25:xix). 


43 


This  value  is  tested  against  a tabulated  value  of  JCifor  degrees  of 

freedom  n-1.  IfX*  > X*  , it  is  concluded  that  there  is  significant 

a>k  T .frfe 

agreement.  Use  table  2 or  complete  tables  from  any  statistics  book. 

7.  Determine  rank  order  by  ordering  the  objectives  on  the  basis  of  the 
Rj's.  Those  with  the  lower  sum  (or  higher  sum,  if  they  were  ranked 
initially  with  higher  numerical  values  being  assigned  to  the  most 
preferred)  are  considered  to  be  the  most  preferred. 

Arrow's  Method  of  Majority  Algorithm. 

Steps  1 through  6 are  identical  to  those  in  the  Football  Poll 
Algorithm. 

7.  Generate  a matrix,  n x n,  in  which  each  of  the  objectives  is  con- 
sidered one  at  a time  and  the  number  cr  times  it  was  preferred  to  each 
of  the  other  objectives  is  counted.  The  objective  considered  is  listed 
on  the  left  of  the  table  and  those  with  which  it  is  considered  are  listed 
across  the  top.  The  array  indicates  how  many  times  the  objectives  on  the 
left  of  the  table  are  preferred  to  the  objectives  with  which  they  are 
compared.  In  the  case  of  ties,  the  single  vote  is  split  and  one-half 
vote  is  awarded  to  each  objective  for  every  pair  of  objectives  tied.  If 
three  or  more  objectives  are  ti^d,  they  still  receive  one -half  votes  in 
pair-wise  comparisons.  The  sum  of  the  complement  cells  in  the  array  will 
equal  the  number  of  judges. 

8.  Generate  another  n x n matrix  in  which  the  number  in  the  dominating 
cell  of  the  previous  matrix  is  replaced  by  a 1 and  the  number  in  the 
complement  by  a 0.  Sum  across  the  rows  for  each  objective,  obtaining  the 
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number  of  objectives  assigned  lower  numerical  values  (higher  preference) 
by  majority  vote.  Those  with  the  higher  sum  being  highest  preferred. 


f 


To  illustrate  these  algorithms,  an  example  opinion  poll  is  now 
provided  and  the  joint  order  is  ranked  using  the  Football  Poll  and 
Arrow's  Method  of  Majority. 

Example  problem. 

Rankings  of  six  judges  of  four  objectives 

(1)  B>AX>D 

(2)  A>B>D>C 

(3)  A>D>B>C 

(4)  B>A>D>C 

(5)  A~B>D>C 

(6)  B>A>OD 


Casting  these  judgments  in  a matrix  of  rankings: 
n 


m 

A 

B 

C 

D 

1 

2.0 

1.0 

3.5 

3.5 

2 

1.0 

2.0 

4.0 

3.0 

3 

1.0 

3.0 

4.0 

2.0 

4 

2.0 

1.0 

4.0 

3.0 

5 

1.5 

1.5 

4.0 

3.0 

6 

2.0 

1.0 

3.5 

3.5 

R. 

9.5 

9.5 

23.0 

18.0 
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Apply  Kendall's  Coefficient  of  Concordance: 


Sum  the  R. 's: 
J 


Z R S = 9.5  * 9.5  +23.0  + 18.0  = 60.0 

J 


Mean  of  the  R. ' s : 

3 


60 

* T z IS 


Obtain^  • 


xL  - (9  5-/5)*+  (9.5  - /S)  * + (Z3  - IS)  * 

+ (te-/s)z  - /33.S 


Tf\ 

Obtain 


\-T3-T4-0 


(Z3- 2. 


T,  --  --  0.5 

A /a 

T,  = (?3'/2  = 0.5- 

o 1 2. 


V 

Z T.  = O.sS-  + 0.5  + O.  5 * l.s 


Calculate  W: 


w * 


(rz)  ™ (n  ' o ) - rn  2 ~T 

133.5 

(iiY6)*(43-4)-  (6)(I  5 


= 0.7807 
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■ 


Where,  Oi'W^l,  and  0 indicates  perfect  disagreement 

1 indicates  perfect  agreement. 

Testing  for  significance: 

Since  n£7  we  actually  tested  against  the  critical  value  from 
Kendall's  Coefficient  of  Concordance. 


A 

C aJc 


133. S 


TtL-ble  1 


- 75.  7 


(rr\  = 6,  n ~ 4) 

Therefore,  since^  > ^ , conclude  there  is  a significant 

c*/c  T*Mc 

agreement  of  the  judgments. 

Having  passed  Kendall's  test  of  agreement,  we  are  prepared  to  proceed 
directly  into  either  of  two  techniques  of  ordering  and  to  proceed 
with  a test  of  normality  for  a third. 


At  this  point  we  can  make  a Football  Poll  ranking  by  merely  ordering 

the  R . 1 s : 

J 

A ~ B > D > C. 


The  ranking  above  concluded  the  sequence  of  operations  for  the 
Football  Poll.  Proceeding  with  Arrow's  Majority  Rule 
Method : 

Generate  a matrix  of  the  times  one  option  is  preferred  to  another 
option: 
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A - 2.5  6 6 2 

B 3.5  6 5 3 

C 0 0 - 1 0 

D 0 1 5 - 1 

Generate  a matrix  for  the  greater  number  of  preferences  of  one  objective 
in  comparison  with  another  objective: 

A B C D 

A - 0 1 12 

B 1 - 1 13 

CO  0 - 0 0 

DO  0 1 1 

At  this  point  we  can  make  an  Arrow's  Majority  Rule  ranking  of  the 
objectives : 

B > A > D > C 

Note  that  the  two  techniques  gave  approximately  the  same  joint  order. 

The  Football  Poll  saw  indifference  between  A and  B,  while  Arrow's 
Majority  Rule  saw  a preference  for  B.  The  reason  for  this  difference  is 
the  joint  order  formed  from  the  Football  Poll  is  influenced  by  whether 
the  judges  like  an  objective  a great  deal  more  than  another  objective, 
while  Arrow's  Majority  Rule  looks  instead  at  how  many  judges  liked  an 


option. 
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APPENDIX  C 


WEIGHTED  METHOD  OF  MAJORITY  PROCEDURE 

Modification  of  Arrow's  Method  of  Majority  Algorithm. 

1.  Two  assumptions  are  necessary  in  order  to  modify  rankings  based 
upon  weighting  the  opinion  of  the  judges  polled  in  relation  to  their 
degree  of  expertise  concerning  the  various  objectives  being  ranked. 

a.  Assumption  1 - The  judge's  ability  to  accurately  establish  his 
preference  for  one  objective  over  another  is  directly  related  to  the 
least  amount  of  knowledge  he  possesses  concerning  either  objective. 

He  will  prefer  (or  not  prefer)  a particular  objective  of  which  he  has 
considerable  knowledge  as  opposed  to  an  unknown  objective  based 
primarily  upon  his  lack  of  appreciation  of  the  unknown  objective. 

b.  Assumption  2 - The  principal  decision  maker  can  assign  a 
subjective  weight  to  the  level  of  knowledge  or  expertise  of  the  judges. 
In  other  words,  he  can  assign  a numerical  value,  possibly  based  upon 
information  provided  by  the  judges  themselves,  between  0 and  1 to 
represent  how  much  he  will  value  the  more  expert  of  the  judges  over  the 
less  expert. 

2.  Based  upon  the  assumptions  listed  in  step  1,  the  principal  decision 
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maker  determines  the  number  of  levels  of  expertise  he  will  consider  (c), 
and  assigns  a numerical  weight  between  0 and  1 to  each  level  (W-^,  where 

L = 1,2, ,c).  In  addition  to  each  judge's  rank  order,  an  expertise 

level  from  1 to  c is  established  for  each  of  the  objectives  ranked, 
utilizing  either  a judge's  self-rating  as  a measure  or  a decision  maker's 
rating  as  a measure.  As  an  alternative  to  these  measures  of  expertise, 
the  judge's  peers  might  rate  him. 

3.  Arrow's  Method  of  Majority  algorithm,  (see  Appendix  B),  is  modified 
by  substituting  the  following  for  step  7: 

For  each  of  the  m judge's  rank  orders,  establish  an  n x n preference 
matrix  A^. , where  n is  the  number  of  objectives  being  ranked,  k = 

1,2, m is  the  particular  judge  and  a^j  = W^,  if  objective  i is 

preferred  to  objective  j (where  L is  the  lowest  level  of  expertise, 
comparing  levels  of  i and  j)  and  0 otherwise.  If  objective  i is  equal 

to  objective  j then  = W^/2.  Generate  a group  frequency  matrix  B 

rr» 

by  summing  these  matrices  as  £ 

K=l 

number  of  judges  that  prefer  objective  i to  objective  j. 

Example  problem. 

Consider  the  earlier  problem,  but  now  assume  the  six  judges,  although 
"expert"  in  the  general  area  of  concern,  are  not  equally  conversant 
with  each  of  the  four  specific  technical  objective  areas.  The  decision 
maker  then  asks  them  to  indicate  whether  they  consider  themselves  to  be: 


A. 


; thus,  each  b^j  will  represent  the 
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fr. 


(1)  vaguely  familiar,  (2)  knowledgeable,  or  (3)  expert  in  each  of 

the  objective  areas.  The  results  of  their  rankings  and  levels  of 

expertise  are  as  follows: 

(1)  Rank  order  B>A>C'V'D 

Expertise  2213 

(2)  Rank  order  A>B>D>C 

Expertise  3212 

(3)  Rank  order  A>D>B^C 

Expertise  2311 

(4)  Rank  order  B>A>D>C 

Expertise  3221 

(5)  Rank  order  A^B>D>C 

Expertise  3312 

(6)  Rank  order  B>A>C*''D 

Expertise  2133 

The  decision  maker  must  weight  the  levels  of  expertise. 

Assume  he  does  so  as  follows: 


Expertise  Level 

Weight 

1. 

Vaguely  familiar 

0.2 

2. 

Knowledgeable 

0.6 

3. 

Expert 

1.0 

Preference  matrices  are  established  for  each  judge: 


1 


51 


JUDGE  3 A B C D 

A - .2  .2  .6 

B 0 - .2  0 

C 0 0-0 

D 0 .2  .2  - 

JUDGE  5 A B C D 

A - .5  .6  .2 

B .5  - .6  .2 

C 0 0-0 

D 0 0 .2  - 


JUDGE  4 A B C D 

A - 0 .2  .6 

B .6  - .2  .6 

C 0 0-0 

D 0 0 .2  - 

JUDGE  6 A B C D 

A - 0 .2  .2 

B .2  - .6  .6 

C 0 0 - .5 

D 0 0 .5  - 
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The  group  frequency  matrix  is  generated: 


The  group  preference  matrix  is  generated: 


The  final  weighted  group  ranking  is  B>A>D^C,  which  in 
this  case  is  identical  to  the  Arrow's  Method  of  Majority 
unweighted  group  rank  order. 
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appendix  d 

TllURSTONE'S  LAW  OF  COMPARATIVE  JUDGMENT 

Algorithm  for  Thurstone's  Law  (Cases  2 through  5). 

Steps  1 through  8 of  Arrow's  Method  of  Majority  Algorithm,  (see 
Appendix  B),  are  utilized,  if,  after  step  6,  a Chi-Square  Goodness 
of  Fit  Test  indicates  the  numerical  values  reflecting  the  judge's 
preference  for  different  objectives  are  normally  distributed. 

After  step  6, a Chi-Square  Goodness  of  Fit  Test,  described  as  follows, 
must  be  inserted  in  the  sequence  which  is  otherwise  identical  to  the 
steps  in  Arrow's  Method  of  Majority: 

The  null  hypothesis  that  the  observed  values  are  not  different  from 
normally  distributed  values  is  tested  by  computing  the  Chi-Square 
test  statistic  from  the  relation 

X4  = 2.  (*•  - mp.  )*/ np.  ) 

Where,  = observed  frequency  of  values  in 
the  i-th  interval 

m = total  number  of  values  in  the  set 
Pi  =*  the  probability  that  a given  value 
will  occur  in  the  i-th  interval 
k = total  number  of  intervals  over  the 
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1 

range  of  the  data; 

thus,  mp  = expected  frequency  of  values 
i 

in  the  i-th  interval. 

X c*/tis  compared  with  ^.3  J f ) 

*(0ne  degree  of  freedom  is  lost  in  summing  the  frequencies  over  k 
intervals,  and  two  are  lost  by  the  requirement  to  estimate  the  mean 
and  variance  of  the  data  set.) 

If  the  null  hypothesis  is  rejected,  and  it  is 

concluded  the  distribution  of  the  joint  order  is  not  normal. 

If  the  null  hypothesis  is  not  rejected,  and  it  is 

concluded  that  there  is  no  reason  to  believe  the  data  is  derived 
from  other  than  a normal  distribution.  In  general,  the  number  of 
judges  exceeds  the  number  of  objectives  being  ranked.  The  Chi-Square 
C70odness  of  Fit  Test  is  applied  to  the  set  of  data  corresponding  to 
each  objective  over  the  range  of  all  judges.  For  instance,  in  this 
example,  there  are  four  objectives,  A through  D.  A separate  test 
would  be  applied  to  each  of  the  four  data  sets.  It  has  been  found 
in  practice  (29:44)  that  sets  at  the  extreme  ends  of  the  joint 
order  are  often  non-normal,  while  the  middle  sets  are  normal.  This 
does  not  invalidate  the  amalgamation  process,  and  is  merely  a reflection 
of  some  distortion  introduced  through  truncation  in  transforming  the 


rankings  to  ordinal  values  and  real  numbers.  When  there  are  a 
relatively  large  number  of  objectives  being  ranked,  an  exception  to 
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the  usual  procedure  may  be  performed.  In  this  case,  where  there  are  at 


least  20  objectives  which  have  been  ranked,  the  date  set  of  Rj's  may 
itself  be  tested  for  normality.  When  this  is  done,  n is  substituted  for 
m in  the  relation.  No  matter  which  procedure  is  used,  the  level  of  con- 
fidence OC  (Greek  letter  Alpha)  and  the  number  of  intervals  k must  be 
selected  arbitrarily  before  conducting  the  test.  In  practice,  a sensi- 
tivity analysis  is  conducted  a priori  to  determine  an  appropriate  level 
of  confidence.  The  number  of  intervals  will  be  selected  so  that  the 
expected  number  or  frequency  of  values  in  the  i-th  interval,  either  i..pp 
or  np^ , will  be  at  least  5 and  preferably  be  10.  (For  example,  if  there 
are  60  judges,  k could  be  chosen  as  6,  and  since  m is  60,  then  mp^=  10  and 
PL  = .1667.  Because  the  degrees  of  freedom  lost  is  three,  the  number  of 
intervals  must  be  at  least  four.  Most  statistical  texts  discuss  this  test 
in  considerable  detail.  For  instance  Guenther  (8:185-87)  covers  this 
test  in  a clear,  concise  manner. 

If  Case  1 applies,  only  the  ordering  need  be  accomplished  and  the 
algorithm  may  commence  at  this  point  without  the  statistical  tests 
being  performed. 

The  steps  for  using  Case  2 will  now  be  listed,  and  then  comments  will 
be  made  concerning  use  of  Cases  1,  3,  4,  and  5. 

Steps  1-8,  plus  a test  for  normality  having  been  accomplished, 
the  following  should  be  done: 


9.  Calculate  the  probabilities  of  differences  that  exist  between  the 


adjacent  objectives,  as  ordered  in  step  8,  using  the  relation 

pC-i  > j ) ~ f (■*- > j [f  (*-  >j)  *■  f ( j > < ) J ■ 

10.  Obtain  z values  from  a table  of  the  standard  normal  distribution 
from  any  statistical  text  book. 

11.  Calculate  the  unbiased  estimate  of  variance  for  the  m values  of 
the  i-th  objective  using  the  relation 

***  ['/<m-  •>]  | (**-  * >, 

where  x is  the  value  of  the  i-th  obiective 
lk  J 

as  ranked  by  the  k-th  judge,  and  xi  is  the 
mean  of  the  values  from  m judges. 

12.  Calculate  the  coefficient  of  correlation  r—,  expressing  the 
degree  of  association  between  the  i-th  and  j-th  objectives,  by  using 
the  relation 

rii  = cov  ./sisi> 

j i j -i-  a 

where  cov^j,  the  covariance,  is  computed  from 
the  equation 

13.  Calculate  the  difference  of  values  between  pairs,  v^  “ vj> 
using  the  complete  form  of  Thurstone's  equation 


Y.  - V.  - £.  .i  s*  ♦ S;  ’ - 2 r s. 
-*■  J V J *J 


m 


where  v.  and  v.  are  the  values  of  the 
1 J 

i-th  and  j-th  objectives. 

14.  Set  the  value  of  the  lowest  ranked  objective  equal  to  1.00,  where 
such  assignment  of  this  value  to  the  least  preferred  objective  is 
arbitrary.  The  value  could  have  been  0,  10,  or  100,  but  1.00  is  a 
convenient  convention. 

15.  Add  the  difference  in  value  to  each  ascending  objective. 

16.  Normalize  the  assigned  values  using  the  relation 


where  i^  is  the  normalized  value  of  the 
i-th  objective,  i is  the  assigned  value 

a n 

of  the  i-th  objective,  and  £ is  the 
sum  of  the  assigned  values  over  the  set  of 
n objectives. 

17.  Now  that  the  values  are  on  the  same  scale,  rank  them 
cardinally. 


Looking  at  the  other  four  cases,  we  see  the  following: 

Case  1 - The  equation  is  used  in  its  complete  form.  This  case  does  not 
apply  to  group  opinion;  it  is  used  only  for  a single  judge  making 
multiple  rankings. 

Case  3 - The  equation  can  be  simplified  by  dropping  the  last  term 

under  the  square  root  if  the  i-th  and  j-th  distributions  are  uncorrelated, 


r. . = 0.  This  may  be  verified  using  the  test  statistic  t computed  from 
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the  relation 


t = r. . V - 2 / V / - r • 

■<7  / J 

The  calculated  value  of  t is  compared  to  the  tabulated  t,  (see  any 
statistical  text  for  a table  of  the  "Student's"  t distribution),  at  m - 
2 degrees  of  freedom  and  a specified  level  of  significance , 0(  (Greek 
letter  Alpha).  If  the  calculated  value  exceeds  the  tabulated  value,  the 
null  hypothesis  that  the  distributions  are  uncorrelated  is  rejected.  The 
conclusion  is  that  the  distributions  are  correlated,  r^j  f 0.  Conversely, 
if  the  calculated  value  does  not  exceed  the  tabulated  value,  the  null 
hypothesis  is  not  rejected,  r^j  = 0. 

Case  4 - The  equation  reduces  to 

v.  - Mj  - 0.70?  zy  (s.  +Sy). 

The  same  hypothesis  test  to  support  this  case  also  supports  the  even  more 
simplified  version,  Case  5.  Therefore,  this  case  is  not  recommended  for 
use  in  group  opinion  amalgamation. 

Case  5 - The  equation  reduces  to 

*•  J J *j 

The  F - test,  where  the  null  hypothesis  is  that  the  variances  are  equal, 
is  applied  by  computing  the  F statistic  from  the  relation 

F = Si  2/s j2, 

where  the  larger  variance  is  the  numerator. 
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This  calculated  value  of  F is  compared  to  the  tabulated  value,  (see 
any  statistical  text  for  a table  of  the  F distribution)  at  a specified 
level  of  significance,  entering  the  table  at  iik  - 1 (column)  and  m j - 1 
(row)  degrees  of  freedom.  If  the  calculated  F exceeds  the  tabulated 
value,  the  null  hypothesis  is  rejected.  The  conclusion  is  that  the 
variances  are  not  equal.  If  the  calculated  value  of  F does  not  exceed 
the  tabulated  value,  the  null  hypothesis  is  not  rejected.  The  conclusion 
is  that  the  variances  are  equal. 

Example  problem. 

Since  the  meaningful  comparison  of  techniques  can  be  made  using 
the  same  problem  for  all  techniques,  and  since  it  is  necessary  that  it 
be  designed  with  a relatively  small  number  of  judges  and  objectives  for  ease 
of  manipulation  and  understanding;  the  example  we  are  using  should  not  pass  a 
normality  test.  This  step  is  omitted,  and  we  will  proceed  directly  from 
the  joint  order  we  obtained  in  the  Arrow's  example  to  step  9 in  our 
sequence  of  operations. 

The  joint  order  is 

B > A > D > C . 

Writing  the  set  of  adjacent  differences: 

(B>A),  (A  > D) , and  (D>C). 

The  probabilities  that  the  i-th  objective  is  preferred  to  the  j-th 
objective  is  calculated: 
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f>  (6  > A)  - 

p (A  >»)  = 


/ ( 3 > A ) 

f (B?A  ) + f(A  >B) 

O.  5833 
6 _ . 

6*0  t'O 


3.5 

3.5  •*  2-5 


/>(*>  C>-  fj-  - 0.8333 


Using  the  table  of  the  standard  normal  distribution,  the  value  of 
the  standard  normal  deviate,  Z.  , corresponding  to  the  p (i>j),  is 
found. 

03,03 

*,*>»•  33999 

^ ( £>  > C)  ~ ^60 

Now  we  need  an  unbiased  estimate  of  standard  deviation. 

i - r //  ,\lr/  - \ 2 


V nr  = i • 5833 

= [U-  i.s®)*  4.  Ci  - i-s-e)1* + (2  - i. 5 


- o.  24/6 

The  other  variances  are  approximately 

Sg1*  0.6416 

Sc2  = O.  0666 
5d*  = O-  3000 


cov  = — 1 

ab 


£ (.*«-  **>(*■*-  * a ^ 


rr»  - | 


- [(2-  1.5-eX'  - '-58) 

•i-  (2. - i.saX'  - 1 • se)3  /«5r  = o.3ii4 


S = tTs7  ' i 0.24/6  - 0.49/S 


- 1 0.64/6  = 0.S0O9 

= covfi  /s4s8  — 

= O.  7903 


5 5 t 0.3H4 

4 0 ~ (O.49is)(0.8c>o9) 


Vv-  V 

=■  0.2/03  f 0.64/6  -h  O.Z4/6  - 2 fo. 7905^).49/S^b. 0oo9^ 
- O.  1072 


The  other  difference  of  values  are  approximately: 

v4-vc-  /.34sa 


V - V - O.  6598 

o c 


Setting 

C = 

1.0000 

Normalizing:  C = 

.1138 

then 

D = 

1.6598 

D = 

.1890 

A = 

3.0096 

A = 

.3425 

B = 

3.1168 

B = 

.3547 

z = 

8.7862 

z = 

1 . 0000 

Which  now 

permits 

cardinal 

ranking  as  displayed  on 

table  3. 
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TABLE  3 

■»  PREFERENCE 

CARDINAL  RANKING  OF  JOINT  ORDER  PREFERENCE 
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APPENDIX  E 
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OTHER  AMALGAMATING  TECHNIQUES 

Tlie  following  techniques  for  nmal  gama  tin>  opinion  were  ioim  lert 
but  rejected  because  they  are  less  desirable  than  those  described  in 
Appendices  B through  D for  accomplishing  similar  purposes. 

Analysis  of  Variance  (16:261-307),  usually  known  imply  as  ANOVA, 
is  customarily  used  in  statistical  analyses  to  detii  « ! tin-  oh  erved 

differences  among  sample  means,  where  more  that  t • j . in  •<  i 
considered,  is  attributable  to  chance.  In  other  or  , . t u a : to 
show  whether  differences  among  several  means  rt  >t  t > < 

specified  level  of  confidence.  By  employing  tht  Dun<  m Mul t i pie -Range 
Test  (16:279-281),  it  is  possible  to  extend  the  ANOVA  to  indicate 
whether  a given  mean  (or  group  of  means),  differs  significantly  from 
another  given  mean  (or  group  of  means).  It  is  then  possible  to  rank 
order  objectives  based  upon  the  ordering  of  judges.  This  technique  is 
rejected  because  it  is  too  laborious.  It  is  also  quite  restrictive. 
Specifically,  the  joint  order  of  the  data  must  be  normal  with  equal 
variances  among  the  rankings. 

A technique  known  as  The  Churchman  and  Ackoff  Approximate  Measure  of 
Value  (5:141-53)  provides  a collective  decision  by  means  of  a majority 
rule  on  a succession  of  comparisons.  The  judges  initially  rank  all 
objectives,  and  an  arbitrary  value  of  100  is  assigned  the  most  preferred, 

with  decreasing  numerical  values  assigned  the  remaining  ordered  objectives. 
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The  judge  is  then  asked  to  compare  the  effect  of  the  initial  most  preferred 
objective  with  that  of  the  combined  effect  of  varying  combinations  of  the 
remaining  objectives.  A decision  as  to  whether  this  objective  is  better, 
equal  to,  or  worse  than  the  combinations  is  made  for  each  possible  combination 
The  majority  vote  is  then  registered.  Adjustments  are  then  made  as 
necessary  in  the  rank  order  and  the  values  ascribed  to  the  objectives. 

This  technique  becomes  extremely  laborious  for  more  than  four  or  five 
objectives  because  of  the  large  number  of  possible  combinations. 

Churchman's  Method  is  rejected  because  of  the  large  requirement  it  imposes 
not  only  on  the  decision  maker,  but  also  on  the  individual  judges. 

There  have  been  a number  of  authors  who  have  recommended  probabilistic 
approaches.  Two  of  the  most  applicable  of  these  are  Winkler  (28:B61-B75) 
and  Raiffa  (17:104-56).  Despite  their  arguments  that  subjective  pro- 
babilities can  be  derived  which  assign  numerical  quantities  to  the 
intuitive  feelings  a judge  has  for  the  occurence  of  some  event,  many  other 
authors  question  the  ease  and  the  reliability  of  this  process.  Svestka 
(35:5-6)  submits  an  argument  against  using  subjective  probabilities, 
stating  that  our  actual  belief  at  any  instant  is  "fugitive  and  variable" 
and  that  it  is  extremely  difficult  to  fix  it  in  a manner  sufficiently 
clear  that  it  may  be  measured.  He  further  elaborates  his  argument 
stating  that  it  is  not  only  difficult  to  ascribe  numbers  to  intensities  of 
feelings,  but  the  things  one  feels  most  strongly  about  he  frequently 
either  takes  for  granted  or  is  totally  unable  to  give  a probability  for. 

The  use  of  a probabilistic  technique  is  rejected  because  of  the  difficulty 
in  directly  assigning  subjective  probabilities. 
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Black's  Single  Peaked  Preferences  (3:74-80)  is  a technique  using 
an  odd  number  of  judges.  Each  judge  orders  the  objectives  individually 
utilizing  a transformation  curve  on  which  there  is  a single  point  for 
which  his  utility  is  maximized  and  his  utility  decreases  in  either 
direction  from  the  optimum.  Taking  the  individual  orderings  representable 
by  single-peaked  preferences,  the  best  joint  order  objective  may  be 
removed  and  then  the  one  best  among  the  remainder,  (the  one  preferred  by  a 
majority  of  the  individuals  to  any  other  objective  remaining),  is  listed 
as  being  second  best,  etc.  Since  this  technique  provides  results  essen- 
tially the  same  as  Arrow's  Method  of  Majority,  but  is  restrictive  as  to 
the  number  of  judges,  requires  transitive  orders,  and  requires  every 
judge  to  order  all  the  alternative  objectives,  it  is  rejected. 

Considered  also  were  Kendall's  Method  of  First  Differences  (29:19-20) 
and  Fishburn's  Ordered  Metric  Method  (29:20-21).  They  are  similar  in  that 


they  both  enable  the  judge  to  cardinally  rank  objectives,  which  he  is 
perhaps  unable  to  rank  directly.  That  is,  instead  of  placing  A,B,C---,n 
in  an  ordered  sequence  and  stating  how  much  more  one  is  preferred  to 
another,  the  judge  orders  the  objectives  and  then  examines  his  preferences 
for  each  adjacent  pair,  ranking  them  in  order  of  their  differences  from 
least  to  greatest  using  Kendall's  Method.  In  Fishburn's  Method,  the 
same  process  is  followed,  but  in  addition,  all  other  possible  differences 
are  listed.  For  example,  Kendall's  method  examines  the  intensity  of 
difference  in  preference  the  judge  feels  between  A and  B,  B and  C,  etc., 
(assuming  for  convenience  of  explanation  that  the  sequential  ordering  and 


natural  alphabetic  lettering  were  identical),  while  Fishburn's  method 
includes  the  adjacent  differences,  but  also  looks  at  A and  C,  A and  D, 
etc.,  where  each  of  the  new  differences  spans  an  interval  exceeding  at 
least  one  of  the  adjacent  differences  already  in  the  ranked  set.  Each  new 
difference  is  compared  to  the  next  greatest  difference  in  the  ranked  set 
until  it  is  observed  to  be  less  than  some  difference.  The  new  difference 
is  placed  into  the  ranked  set  so  that  it  is  greater  than  the  difference 
below  it  and  less  than  the  differences  above  it.  This  process  continues 
until  all  possible  differences  are  ranked.  When  the  judges  have  completed 
their  ranking  of  objectives  and  of  differences,  the  decision  maker  then 
assigns  values  to  the  spatial  differences  compatible  with  the  relationships 
stated  by  the  judges.  The  values  of  the  differences  are  then  used  to 
compute  the  values  of  the  objectives.  The  objective  values  are  then 
normalized.  Kendall's  Method  of  First  Differences  appears  to  be  a feasible 
technique;  however,  Thurstone's  Law  of  Comparative  Judgements  accomplishes 
the  same  result,  is  more  amenable  to  large  numbers  of  judges  and  objectives, 
and  is  more  compatible  with  the  otiier  techniques  selected.  If  cardinal 
ranking  is  an  overwhelming  requisite  and  if  the  joint  order  fails  to  meet 
the  normality  requirement,  Kendall's  Method  of  First  Differences  might  be 
considered.  Fishburn's  Higher  Ordered  Metric  Measure  is  similar  to  the 
Churchman  and  Ackoff  Approximate  Method  considered  earlier,  in  that,  for 
other  than  very  minimal  numbers  of  objectives  it  becomes  too  laborious  for 
both  the  judges  and  the  decision  maker. 


Certainly  this  paper  would  be  remiss  if  it  did  not  address  the  DELPHI 
Technique.  It  was  developed  to  obtain  the  benefits  accruing  from  an 


interplay  of  ideas  between  a group  of  judges  without  being  plagued  with 
the  attendant  problem  of  staff  meetings,  conferences,  panels,  or  brain- 
storming sessions.  Its  developers  were  attempting  to  avoid  the  pitfalls 
which  may  be  associated  with  such  group  gatherings  in  their  design  of 
DELPHI.  Inequality  of  the  judges  because  of  rank  or  experience  and 
psychological  factors  such  as  a dominant  personality,  the  desire  for 
group  approval,  or  a stubborn  unwillingness  to  change  from  a previously 
expressed  opinion  are  some  of  the  alleged  shortcomings  of  group  meetings 
they  attempted  to  design  out  of  the  DELPHI  Technique  (30:7  & 11:2). 
Originally  developed  by  the  Rand  Corporation  in  the  late  1940's  (11:2-4), 
it  was  widely  used  by  universities,  by  the  military  services,  by  the 
Defense  Intelligence  Agency  and  the  Central  Intelligence  Agency,  and  within 
civilian  industry.  It  replaces  direct  debate  with  a carefully  designed 
program  of  sequential  individual  interrogations  (4:2-6).  The  judges 
receive  information  on  the  previous  response  which  includes  the  median  and 
the  interquartile  range  of  those  rankings.  They  may  also  be  provided  one, 
or  a few,  relevant  facts  in  addition  to  statistical  feedback  alone 
(6:28-30).  An  obvious  limitation  to  the  use  of  DELPHI  is  that  the  very 
nature  of  the  feedback  process  necessitates  the  availability  of  considerable 
time.  Sackman,  in  an  indepth  assessment  of  DELPHI  (19 :67-68)>  concludes 
that  it  is  "basically  an  unreliable  and  scientifically  unvalidated  technique 
in  principal"  and  that  "its  claim  to  represent  valid  expert  opinion  is 

empirically  untenable  and  overstated."  He  further  concluded  that  the 
allegedly  designed-in  features  of  the  superiority  of  remote  and  private 


opinion  over  face-to-face  encounter,  as  well  as  their  counter-statements, 
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are  unproven  generalizations,  and  DELPHI  anonymity  reinforces  unaccount- 
ability in  method  and  findings.  It  also  "discourages  adversary  process 
and  inhibits  exploratory  thinking."  Consequently,  it  was  discarded  as 
a recommended  technique. 
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